The Design and Operation of The Keck Observatory Archive

The Infrared Processing and Analysis Center (IPAC) and the W. M. Keck Observatory (WMKO) operate an archive for the Keck Observatory. At the end of 2013, KOA completed the ingestion of data from all eight active observatory instruments. KOA will continue to ingest all newly obtained observations, at an anticipated volume of 4 TB per year. The data are transmitted electronically from WMKO to IPAC for storage and curation. Access to data is governed by a data use policy, and approximately two-thirds of the data in the archive are public.Comment: 12 pages, 4 figs, 4 tables. Presented at Software and Cyberinfrastructure for Astronomy III, SPIE Astronomical Telescopes + Instrumentation 2014. June 2014, Montreal, Canad

Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking

Montage is a portable software toolkit for constructing custom, science-grade mosaics by composing multiple astronomical images. The mosaics constructed by Montage preserve the astrometry (position) and photometry (intensity) of the sources in the input images. The mosaic to be constructed is specified by the user in terms of a set of parameters, including dataset and wavelength to be used, location and size on the sky, coordinate system and projection, and spatial sampling rate. Many astronomical datasets are massive, and are stored in distributed archives that are, in most cases, remote with respect to the available computational resources. Montage can be run on both single- and multi-processor computers, including clusters and grids. Standard grid tools are used to run Montage in the case where the data or computers used to construct a mosaic are located remotely on the Internet. This paper describes the architecture, algorithms, and usage of Montage as both a software toolkit and as a grid portal. Timing results are provided to show how Montage performance scales with number of processors on a cluster computer. In addition, we compare the performance of two methods of running Montage in parallel on a grid.Comment: 16 pages, 11 figure

Montage: An Astronomical Image Mosaic Service for the NVO

Montage is a software system for generating astronomical image mosaics according to user-specified size, rotation, WCS-compliant projection and coordinate system, with background modeling and rectification capabilities. Its architecture has been described in the proceedings of ADASS XII and XIII (Berriman et al. 2003, 2004). It has been designed as a toolkit, with independent modules for image reprojection, background rectification and co-addition, and will run on workstations, clusters and grids. The primary limitation of Montage thus far has been in the projection algorithm. It uses a spherical trigonometry approach that is general at the expense of speed. The reprojection algorithm has now been made 30 times faster for commonly used tangent plane to tangent plane reprojections that cover up to several square degrees, through modification of a custom algorithm first derived for the Spitzer Space Telescope. This focus session will describe this algorithm, demonstrate the generation of mosaics in real time, and describe applications of the software. In particular, we will highlight one case study which shows how Montage is supporting the generation of science-grade mosaics of images measured with the Infrared Array Camera aboard the Spitzer Space Telescope

The NASA/IPAC Infrared Science Archive (IRSA): The Demo

This paper describes the services available at the NASA/IPAC Infrared Science Archive (IRSA). Currently there are nearly 250,000 data requests a month, taking advantage of IRSA's data repository which includes 660 million sources (60 catalogs), 10 million images (22 image sets; 10.4 TB) and over 30,000 spectra (7 spectroscopic datasets). These data are the science products of: The Two Micron All Sky Survey (2MASS), The Infrared Astronomical Satellite (IRAS), The Midcourse Space Experiment (MSX), The Submillimeter Wave Astronomy Satellite (SWAS), The Infrared Space Observatory (ISO), The Infrared Telescope in Space (IRTS), The Spitzer First Look Survey (FLS), Spitzer Legacy & Ancillary data, Spitzer Reserved Observations (ROC) and the Spitzer Space Telescope data. IRSA is also seamlessly interoperable with ten remote archives and services: GOODS, ISO, MAST, VizieR, DSS, NVSS, FIRST, HEASARC, NED and JPL, which help expand the available dataset wavelength range from X-ray to radio. The majority of IRSA's image collections are Simple Image Access (SIA) compliant and are available through the Virtual Observatory (VO) data mining tools. The IRSA demo includes IRSA's ¯ve main services: inventory service RADAR, catalog query service Gator, data fusion service OASIS, general search service for complex data collections Atlas, and IRSA's 2MASS Image data access services. IRSA's website is http://irsa.ipac.caltech.edu

A case study in adaptable and reusable infrastructure at the Keck Observatory Archive: VO interfaces, moving targets, and more

The Keck Observatory Archive (KOA) (https://koa.ipac.caltech.edu) curates all observations acquired at the W. M. Keck Observatory (WMKO) since it began operations in 1994, including data from eight active instruments and two decommissioned instruments. The archive is a collaboration between WMKO and the NASA Exoplanet Science Institute (NExScI). Since its inception in 2004, the science information system used at KOA has adopted an architectural approach that emphasizes software re-use and adaptability. This paper describes how KOA is currently leveraging and extending open source software components to develop new services and to support delivery of a complete set of instrument metadata, which will enable more sophisticated and extensive queries than currently possible. In August 2015, KOA deployed a program interface to discover public data from all instruments equipped with an imaging mode. The interface complies with version 2 of the Simple Imaging Access Protocol (SIAP), under development by the International Virtual Observatory Alliance (IVOA), which defines a standard mechanism for discovering images through spatial queries. The heart of the KOA service is an R-tree-based, database-indexing mechanism prototyped by the Virtual Astronomical Observatory (VAO) and further developed by the Montage Image Mosaic project, designed to provide fast access to large imaging data sets as a first step in creating wide-area image mosaics (such as mosaics of subsets of the 4.7 million images of the SDSS DR9 release). The KOA service uses the results of the spatial R-tree search to create an SQLite data database for further relational filtering. The service uses a JSON configuration file to describe the association between instrument parameters and the service query parameters, and to make it applicable beyond the Keck instruments. The images generated at the Keck telescope usually do not encode the image footprints as WCS fields in the FITS file headers. Because SIAP searches are spatial, much of the effort in developing the program interface involved processing the instrument and telescope parameters to understand how accurately we can derive the WCS information for each instrument. This knowledge is now being fed back into the KOA databases as part of a program to include complete metadata information for all imaging observations. The R-tree program was itself extended to support temporal (in addition to spatial) indexing, in response to requests from the planetary science community for a search engine to discover observations of Solar System objects. With this 3D-indexing scheme, the service performs very fast time and spatial matches between the target ephemerides, obtained from the JPL SPICE service. Our experiments indicate these matches can be more than 100 times faster than when separating temporal and spatial searches. Images of the tracks of the moving targets, overlaid with the image footprints, are computed with a new command-line visualization tool, mViewer, released with the Montage distribution. The service is currently in test and will be released in late summer 2016

The Automated Palomar 60-Inch Telescope

We have converted the Palomar 60-inch telescope (P60) from a classical night assistant-operated telescope to a fully robotic facility. The automated system, which has been operational since September 2004, is designed for moderately fast (t <~ 3 minutes) and sustained (R <~ 23 mag) observations of gamma-ray burst afterglows and other transient events. Routine queue-scheduled observations can be interrupted in response to electronic notification of transient events. An automated pipeline reduces data in real-time, which is then stored on a searchable web-based archive for ease of distribution. We describe here the design requirements, hardware and software upgrades, and lessons learned from roboticization. We present an overview of the current system performance as well as plans for future upgrades.Comment: Accepted in PASP; 26 pages, 7 figures; high resolution version at http://www.srl.caltech.edu/~cenko/public/papers/p60.p

Pegasus: a framework for mapping complex scientific workflows onto distributed systems

This paper describes the Pegasus framework that can be used to map complex scientific workflows onto distributed resources. Pegasus enables users to represent the workflows at an abstract level without needing to worry about the particulars of the target execution systems. The paper describes general issues in mapping applications and the functionality of Pegasus. We present the results of improving application performance through workflow restructuring